In an image, a transition occurs between the edge of the subject and its background. This transition can be very narrow if the subject is sharply focused. However, lenses are not so sharp, nor subjects and cameras so still, that the transition will occur between two adjacent pixels in a digital image. The transition occurs over a minimum of at least three pixels, and up to the number of pixels comprising the width of the blurred edge of a moving or out-of-focus subject.
In motion picture and video applications, subjects are generally in motion. The resulting semi-transparency of blurred edges must be retained, but the contribution of the initial background seen through these blurred edges must be removed and replaced with the chosen background. Otherwise, any portion of the initial background remaining in the semitransparent edges becomes edge artifacts in the composite image.
The process described in U.S. Pat. No. 6,134,346, which has become known in the graphic arts industry as the Ultimate Knockout process, will be referred to as such in the subsequent discussion. This Knockout process for the graphics arts has reached such a level of perfection that many of those skilled in the art of compositing are not able to detect that the image they observe is a composite. However, Knockout requires two manually drawn boundary lines to define the subject-to-background transition area, and another outline at the perimeter of a shadow. The inner, outer, and shadow boundary lines are hand-drawn, using a cursor, close to the transition area without touching it. The requirement of hand drawn boundary lines has prevented the use of Knockout for motion picture and video applications.
The invention to be described makes possible the extension of the Knockout process to motion picture and video applications by automatically generating the transition area boundary lines, and by providing a modified clear frame containing shadow information.
U.S. Pat. No. 4,800,432 describes a xe2x80x9cVideo Difference Key Generatorxe2x80x9d that employs the subtraction of two images; the first image excludes the subject, while the second image includes the subject. Their difference creates a key signal. A key signal is a switch.
In a switching system there is no proper place to switch, since every pixel in the subject-to-background transition area contains both background and subject. In blurred subject edges, switching creates disturbing edge artifacts in the composite image. Switching does not occur in the Ultimate Knockout image compositing method. The contribution of the background to each pixel in the transition area is determined and removed by subtraction.
While the comparison of two images photographed at different times has been used successfully by military and government agencies to detect changes in the area photographed, it has not achieved much success in real time video because of edge artifacts and other problems. The importance of the difference principle in image compositing is that it does not require a colored backing or uniform illumination to detect a change between two images.
Changes in a current image frame from variations in room light level, and automatic camera adjustments, are determined and incorporated into a clear (reference) frame, thereby resulting in a zero difference in the background region when the clear frame and a current frame are subtracted. Any deviation from zero defines the beginning edge of a transition from background to subject.
The location of the outer edge of a transition area is further refined by subtracting derivatives of the RGB signals in each frame, and by the use of a fourth color channel when available.
The difference of derivitives (dcu-dcl) identifies the subject to background transition area, whose edges define the inner and outer boundary lines. These automatically generated inner and outer boundary lines permit the use of the Ultimate Knockout Process in motion picture and video image compositing.
Shadow areas on the backing are retained and may be transferred to the inserted background scene. Small areas on the backing having the same color and luminance as the subject are prevented from printing through the subject by a second xe2x80x9cderivativexe2x80x9d alpha channel for said small areas.